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rhe  ability  to  quantify  risk  is  essential  to  the  processes  of  budgeting 
and  scheduling.  During  the  process  of  hiring  to  complete  specified 
tasks,  customers  must  be  able  to  verify  contractor  estimates  and  to 
make  sound  judgments  on  the  risks  of  cost  overruns  and  time  delays.  The 
following  questions  are  central  to  this  paper:  Do  developers  with  little  expe¬ 
rience  overestimate  or  underestimate  the  complexity  of  the  task  because  of 
their  experience,  the  assumptions  they  make,  the  models  they  select,  and 
how  they  define  the  model  limits?  What  are  the  sources  of  risk  associated 
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with  project  cost  estimation?  How  can  such  risk  be  quantified?  This  article 
proposes  a  systematic  acquisition  process  aimed  at  assessing  and  managing 
the  risks  of  cost  overruns  and  time  delays  associated  with  software  develop¬ 
ment.  We  propose  an  acquisition  process  of  four  phases  grounded  on  three 
basic  premises:  (a)  Any  single-value  estimate  of  cost  or  completion  time  is 
inadequate  to  capture  and  represent  the  variability  and  uncertainty  associ¬ 
ated  with  cost  and  schedule.  Probabilistic  quantification  is  advocated,  us¬ 
ing,  in  this  paper,  the  fractile  method  and  triangular  distribution,  (b)  The 
common  expected  value,  when  used  as  a  measure  of  risk,  is  inadequate; 
further,  if  used  as  the  sole  measure  of  risk,  it  may  lead  to  inaccurate  results. 
The  conditional  expected  value  of  risk  of  extreme  events  is  adopted  to 
supplement  and  complement  the  common  unconditional  expected  value, 
(c)  Probing  the  sources  of  risks  and  uncertainties  associated  with  cost  over¬ 
runs  and  time  delays  in  software  development  is  essential  for  the  ultimate 
management  of  technical  and  nontechnical  risks.  This  article  is  based  on  a 
technical  report  published  by  the  Software  Engineering  Institute  (Haimes 
and  Chittister,  1993). 


PREFACE 

The  software  development  community  has  not  been  able  to  agree  upon 
a  set  of  measures  to  define  the  basic  building  blocks  that  can  be  used  to 
generate  cost  and  schedule  estimates.  For  example,  in  most  other  engi¬ 
neering  fields,  cost  estimates  are  based  on  basic  measures.  Examples 
include  BTUs,  PSIs,  length  of  experience,  complexity  of  the  require¬ 
ments,  software  language  to  be  used,  and  estimated  number  of  lines  of 
code.  The  relationships  among  these  factors  and  the  cost  or  schedule 
estimate  are  not  always  clear,  and  this  raises  some  questions  as  to  the 
validity  of  the  estimates  in  any  particular  case. 

The  following  quotations  excerpted  from  Innovative  Contracting  Prac¬ 
tices:  Transportation  Research  Circular  No.  386,  published  by  the  Na¬ 
tional  Research  Council  (December  1991)  highlight  the  current  dismal 
state  of  contracting  practices: 

•  Innovative  contracting  techniques  have  been  developed  more  in 
foreign  countries  than  in  the  United  States. 

•  Unfortunately,  the  lowest  initial  cost  may  not  result  in  the  lowest 
overall  cost. 

•  In  fact,  current  contracting  practices  provide  little  incentive  for 
industry  to  be  innovative. 


122  -  Spring  1995 


Acquisition  Review  Quarterly 


An  Acquisition  Process  for  the  Management  of  Nontechnical  Risks 
_ Associated  With  Software  Development 


•  Agencies  should  develop  contractor  responsibility  tests  that  reflect 
quality  and  performance  factors;  these  tests  should  be  examined 
and  possible  modifications  developed. 

•  Indeed,  the  ability  to  assess  quality  and  performance  are  directly 
related  to  the  ability  to  assess  risk. 

•  From  a  Summary  of  Questionnaire  Findings: 

This  [pre-bid  conferences]  concept  was  the  most  popular,  receiv¬ 
ing  a  positive  response  from  more  than  85%  of  the  states  partici¬ 
pating  in  the  survey.  Better  understanding  of  the  scope  of  work, 
reduction  in  unanticipated  construction  conflicts,  plan  revisions, 
and  other  value  engineering  benefits  can  result  from  such  con¬ 
ferences.  Specialty  jobs,  especially  fast-track  projects,  are  most 
appropriate  for  this  process. 

•  Risk  management  and  assurance.  End-result  specifications  and  a 
determination  of  QA  enter  into  this  issue.  Although  not  currently 
being  practiced,  many  agencies  are  considering  this  concept  for 
future  application. 

The  questionnaire  indicated  that  innovation  has  intensified  in  selected 
topic  areas.  Many  agencies  are  implementing  quality  assurance-quality 
control  (QA-QC)  philosophies,  contractor  surveying,  value  engineering, 
off-peak  time  incentives,  alternative  bidding  on  structures,  and  other 
concepts.  Additionally,  cost-saving  and  profitable  concepts  are  being 
considered  for  future  use  and  further  development.  On  the  other  hand, 
many  agencies  expressed  interest  in  receiving  guidelines  on  other  con¬ 
cepts  that  were  less  understood. 

INTRODUCTION 

Three  major  classes  of  likely  adverse  consequences  are  prevalent  in 
software  development:  risk  of  cost  overrun,  risk  of  time  delay  in  the 
completion  schedule,  and  risk  of  not  meeting  performance  specifica¬ 
tions.  Here  risk  is  defined  as  a  measure  of  the  probability  and  severity  of 
adverse  effects  (Lowrance  1976).  The  first  two  risks,  cost  overrun  and 
time  delay,  are  nontechnical  risks  and  the  third,  performance  specifica¬ 
tions,  is  software  technical  risk;  more  precise  definitions  are  found  in 
Chittister  and  Haimcs  (1993).  The  focus  of  this  paper  is  on  the  quantifi¬ 
cation  (assessment)  and  management  of  software  nontechnical  risks, 
such  as  cost  overruns  and  time  delays. 

The  more  central  the  role  that  software  plays  in  overall  system  inte- 
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gration  and  coordination,  the  more  likely  the  impact  of  delivery  delay 
and/or  of  major  cost  overruns.  A  series  of  auditing  studies  conducted  by 
the  General  Accounting  Office  (GAO)  in  1992  reveal  almost  an  across- 
the-board  epidemic  of  cost  overruns  and  time  delays  in  meeting  comple¬ 
tion  schedules  associated  with  software  development  for  selected  gov¬ 
ernment-sponsored  projects.  A  case  in  point  is  the  C-17  aircraft,  cited  in 
the  previously  mentioned  GAO  report,  which  experienced  a  major  cost 
overrun  and  delivery  delay. 

In  spite  of  the  efforts  made  by  some  of  the  Source  Selection  Authori¬ 
ties  (SSAs)  and  by  their  respective  boards  in  selecting  contractors,  the 
Department  of  Defense  (DoD)  has  still  had  serious  software  delays. 
Indeed,  an  SSA  conducts  a  thorough  search,  examining,  among  other 
factors,  the  organizational  capabilities  of  the  contractor  by  evaluating 
performance  history.  In  some  cases  a  set  of  Key  Practice  Areas  (KPAs) 
are  examined,  such  as  the  processes  of  formal  cost  estimation  and  pro¬ 
gram  management,  as  well  as  metrics  for  evaluating  various  performance 
criteria. 

The  Software  Engineering  Institute  (SEI)  at  Carnegie  Mellon  Uni¬ 
versity  has  also  developed  a  methodology  known  as  Software  Capability 
Evaluation  (SCE)  that  (Humphrey  &  Sweet  1987)  used  to  assess  the 
software  engineering  capability  of  contractors.  The  SCE  seeks  to  answer 
the  question:  Can  the  organization  build  the  product  correctly?  It  does 
so  by  considering  three  separate  aspects  of  the  contractor’s  expertise: 

•  organization  and  resource  management; 

•  the  software  engineering  process  and  its  management;  and 

•  available  tools  and  technology. 

Another  tool,  a  risk  taxonomy,  also  developed  at  SEI,  addresses  the 
sources  of  software  technical  risk  and  attempts  to  answer  the  question:  “Is 
the  organization  building  the  right  product?”  (Carr  et  al.  1993).  The  SCE 
and  the  taxonomy,  then,  offer  methods  of  assessing  organizational  pro¬ 
cesses  and  software  technical  risks;  this  article  presents,  on  the  other  hand, 
a  process  for  quantifying  the  risks  of  project  cost  and  schedule  overruns. 

OVERVIEW  OF  THE  CONCEPTUAL  FRAMEWORK 

In  this  article,  we  present  a  methodological  framework  for  selecting  a 
contractor  that  can  assist  the  customer  in  minimizing  the  risks  of  project 
cost  overruns  and  schedule  delays.  Although  factors  other  than  the  se¬ 
lection  of  contractor(s)  may  decisively  affect  software  technical  and  non- 
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technical  risks,  they  are  treated  here  only  as  a  general  background.  See 
Chittister  and  Haimes  (1993,  1994)  for  a  more  in-depth  discussion  of 
these  factors. 

The  process  of  selecting  contractors  is  by  itself  quite  complex.  The 
process  is  driven  by  legal,  organizational,  technical,  financial,  and  other 
considerations — all  of  which  serve  as  sources  of  risk.  Because  the  world 
within  which  software  engineering  developed  is  nondeterministic  and 
the  central  tendency  measure  of  random  events  conceals  vital  and  criti¬ 
cal  information  about  these  random  events,  special  attention  is  focused 
on  the  variance  of  these  events  and  on  their  extremes.  Two  approaches — 
the  fractile  method  and  triangular  distribution — are  adopted  here  to 
quantify  the  probabilities  of  project  cost  overrun  and  delay  in  comple¬ 
tion  schedule.  To  capture  the  range  of  variation  and  the  extremes  of 
these  probabilities,  conditional  expected  values  of  extreme  events  are 
calculated  using  the  partitioned  multiobjective  risk  method  (PMRM) 
(Asbeck  &  Haimes  1984)  to  supplement  the  common  expected  value  of 
software  nontechnical  risk.  To  accomplish  this  objective,  the  fractile 
method,  triangular  distribution,  and  the  PMRM  are  introduced.  Ex¬ 
amples  are  included  to  clarify  the  appropriate  application  of  these  meth¬ 
ods  and  to  demonstrate  their  utility. 

Figure  1  represents  the  thinking  of  the  methodological  framework. 
The  framework  can  be  viewed  with  four  major  phases.  The  purpose  of 
Phase  I  is  to  quantify  the  variances  in  the  contractor’s  cost  and  schedule 
estimates  by  constructing  probability  density  functions  (PDFs)  through 
triangular  distributions,  the  fractile  method,  or  through  any  other  meth¬ 
ods  that  seem  suitable  to  the  contractor.  Extreme  events  are  assessed 
from  these  PDFs.  In  Phase  II,  using  the  SEI  Taxonomy,  interviews,  and 
the  PMRM,  the  sources  of  risks  and  uncertainties  associated  with  each 
contractor  are  probed  and  evaluated.  The  assumptions  and  premises, 
which  provide  the  basis  for  generating  the  variances  in  the  contractor’s 
estimates,  are  identified  and  evaluated;  and  the  conditional  expected 
value  of  risk  of  extreme  cost  overruns  and  time  delays  are  constructed 
and  evaluated.  Phase  III  analyzes,  ranks,  filters,  and  compares  the  sig¬ 
nificance,  interpretation,  and  validity  of  each  contractor’s  assumptions 
and  premises.  And  the  probabilities  of  technical  and  nontechnical  risks 
are  assessed.  In  executing  Phase  III,  three  tools  and  methodologies  are 
used:  (1)  independent  verification  and  validation  team;  (2)  the  risk  rank¬ 
ing  and  filtering  method;  and  (3)  comparative  analysis.  In  the  final  phase, 
Phase  IV,  conclusions  are  drawn  on  the  basis  of  all  the  previously  gener¬ 
ated  evidence,  including  the  opinions  of  expert  judgment.  The  ultimate 
objective  of  the  methodological  approach  is  to  minimize  the  following 
three  objectives  or  indices  of  performance: 
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Figure  1.  Proposed  Acquisition  Process 
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r 

risk  of  project  cost  overrun; 

risk  of  project  completion  time  delay; 

risk  of  not  meeting  performance  criteria. 


Clearly,  multiobjective  trade-off  analysis,  using  for  example,  the  sur¬ 
rogate  worth  trade-off  (SWT)  method,  needs  to  be  conducted  where  all 
costs  and  risks  are  kept  and  trade  off  in  their  units. 

The  objective  of  this  article  is  to  develop  scientifically  sound  and 
pragmatic  answers  to  some  of  the  lingering  problems  and  questions 
concerning  assessment  and  management  of  risks  of  cost  overruns  and 
time  delays  associated  with  software  engineering  development. 

It  is  constructive  to  discuss  the  four-phase  acquisition  process  in  more 
detail: 


1.  Phase  I  will  be  demonstrated  through  the  construction  of  the  prob¬ 
ability  density  functions  (using  the  fractilc  method  and  triangular 
distribution)  and  through  the  assessment  of  extreme  events  (using 
the  partitioned  multiobjective  risk  method)  by  calculating  the  con¬ 
ditional  expected  value  of  extreme  events  to  supplement  the  com¬ 
mon  unconditional  expected  value  of  cost  overrun. 

2.  Phase  II,  through  the  use  of  the  Taxonomy-Based  Questionnaire, 
interviews,  and  the  quantification  of  risk  of  extreme  events,  pro¬ 
vides  a  mechanism  to  probe  the  sources  of  risks  and  uncertainties, 
identify  and  evaluate  the  assumptions  that  have  generated  the  vari¬ 
ances  for  each  bidding  contractor,  and  construct  the  conditional 
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expected  value  of  risk  of  extreme  events,  f4(»).  The  following  dis¬ 
cussion  will  focus  on  probing  the  sources  of  extreme  events  and 
the  contractor’s  attitude: 

Extreme  Events  —  The  shape  of  the  probability  density  function,  and 
particularly  the  behavior  of  the  tail  of  the  distribution,  markedly  influ¬ 
ence  the  conditional  expected  value  of  extreme  events.  To  demonstrate 
the  effect  of  the  tail  of  the  distribution  on  projected  cost  overruns  or 
time  delays,  all  three  examples  have  at  least  one  project  cost  estimate 
with  a  long  tail  (i.e.,  a  major  cost  overrun  albeit  with  a  relatively  low 
probability). 

Most  customers  are  concerned  with  major  cost  overruns  and  time 
delays  of  any  magnitude.  In  other  words,  most  customers  want  to  pre¬ 
vent  disastrous  events  that  are  beyond  point  b  in  Figure  2.  The  region  to 
the  left  of  b  (cost  overruns  that  would  not  exceed  10-15%)  is  commonly 
represented  by  the  expected  value  measure  of  risk,  f5(«),  whereas  the 
region  to  the  right  of  b  is  captured  by  the  conditional  expected  value  of 
risk  of  extreme  events,  f4(»).  With  the  help  of  the  Taxonomy-Based 
Questionnaire  we  can  probe  the  sources  of  uncertainties  and  variabili¬ 
ties  leading  to  f5(»)  and  f4(«).  Indeed,  the  ultimate  efficacy  of  risk  assess¬ 
ment  is  its  management  through  early  identification,  quantification,  and 
prevention.  Such  a  probe  provides  insights  into  the  contractor’s  assump¬ 
tions  about  what  can  go  wrong  in  a  severe  way  that  might  cause  the  risk 


Low  and  Moderate  Severity 

Medium  and  High  Exceedance  Probability 


Figure  2.  Mapping  the  Probability  Partitioning  Onto  the  Damage  Axis 
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of  extreme  cost  overrun  or  time  delay  to  be  catastrophic. 

Contractor’s  Attitude  —  The  Taxonomy-Based  Questionnaire,  along 
with  the  measurements  of  risk  of  cost  overruns  and  time  delays  through 
the  conditional  expected  value  of  risk  of  extreme  events  and  the  uncon¬ 
ditional  (common)  expected  value  of  risk  should  explain  not  only  the 
contractor’s  technical,  financial,  and  other  managerial  assumptions  and 
premises,  but  also  the  contractor’s  attitude  toward  risk.  When  a  contractor’s 
projection  of  lowest,  most  likely,  and  highest  project  costs  falls,  for  example, 
in  a  close  range,  there  are  several  possible  explanations: 

1.  The  contractor  is  a  risk  seeker  (a  risk-averse  contractor  would 
have  projected  a  much  wider  spread  in  the  lowest,  highest,  and 
most  likely  project  cost). 

2.  The  contractor  is  very  knowledgeable  and  thus  has  confidence  in 
the  tight  projections. 

3.  The  contractor  is  ignorant  of  the  major  technical  details  and  com¬ 
plexity  of  the  project’s  specifications;  thus,  inherent  uncertainties 
and  variabilities  associated  with  the  project  have  been  overlooked. 
Otherwise,  the  contractor  would  have  projected  a  wider  spread 
between  the  most  likely  and  highest  cost  projections. 

The  Taxonomy  not  only  constitutes  an  important  instrument  with  which 
to  discover  the  reasons  for  the  uncertainties  and  variabilities  associated 
with  the  contractor’s  projections,  it  also  provides  a  mechanism  that  al¬ 
lows  the  customer  to  assess  the  validity  and  soundness  of  the  contractor’s 
assumptions.  Indeed,  the  Taxonomy-Based  Questionnaire,  which  is  sys¬ 
tematic,  structured,  and  repeatable,  is  an  invaluable  process  with  which 
the  customer  can  elicit  answers  to  the  reasons  for  the  contractors’  vari¬ 
abilities.  The  accumulated  assumptions  on  each  contractor  are  then  com¬ 
pared  and  analyzed. 

In  Phase  III,  an  analysis  and  comparison  of  the  significance  and  valid¬ 
ity  of  the  contractor’s  assumptions  about  technical  and  nontechnical 
risks  is  conducted.  This  is  accomplished  by  an  Independent  Verification 
and  Validation  (IVV)  team,  the  Risk  Ranking  and  Filtering  (RRF) 
method,  and  other  comparative  analysis  methods.  In  comparing  assump¬ 
tions,  a  number  of  issues  must  be  addressed:  stability  and  precedence  of 
the  requirements;  need  for  research  about  solutions;  politics  and  stabil¬ 
ity  of  funding;  overall  knowledge  or  the  lack  thereof;  level  of  experience 
of  key  personnel;  maturity  of  technology;  and  maturity  of  the  organiza¬ 
tion. 
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In  making  these  comparisons,  the  customer  will  look  at  the  reasons 
for  the  assumptions  and  whether  they  are  based  on  knowledge  or  naivete, 
and  whether  the  contractor  has  a  conservative/risk-averse  or  liberal/risk¬ 
seeking  attitude.  These  issues  are  highlighted  in  the  example  problem  in 
subsequent  discussions.  Contractor  A  is  projecting  a  50%  cost  overrun 
as  the  worst  case,  while  Contractor  B  is  projecting  “only”  a  40%  cost 
overrun  as  the  worst  case.  Is  Contractor  A  more  knowledgeable  or  more 
conservative  than  Contractor  B?  Or  does  the  reason  for  this  difference 
lie  elsewhere?  Is  Contractor  A  risk-averse  whle  Contractor  B  is  risk 
seeking?  The  information  generated  by  the  IVV  team,  the  RRF  method, 
and  through  other  comparative  analysis  tools  will  be  subjected  to  the 
expert  judgment  of  the  customer’s  team,  leading  to  Phase  IV  of  the 
proposed  acquisition  process. 

Phase  IV  is  the  completion  step  where  conclusions  are  drawn  based 
on  the  accumulated  evidence.  Expert  judgment  is  used  in  conjunction 
with  multiobjective  trade-off  analysis  methods,  such  as  the  surrogate 
worth  trade-off  (SWT)  method  (Haimes  &  Hall  1974).  Adopting  the 
systemic  proposed  acquisition  process  should  markedly  reduce  the  like¬ 
lihood  of  major  and  catastrophic  technical  and  nontechnical  risks. 

CRITICAL  FACTORS  THAT  AFFECT 
SOFTWARE  NONTECHNICAL  RISK 

The  proposed  methodological  framework  for  the  quantification  and 
management  of  software  nontechnical  risk — the  risk  of  cost  overrun  and 


Figure  3.  Graphical  CDF  for  Project  Cost  Increase  for  Contractor  A 
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time  delay  associated  with  software  development — is  grounded  on  the 
premise  that  such  management  must  be  holistically  based.  A  holistic 
approach  requires  complete  accounting  of  all  important  and  relevant 
forces  that  drive  the  dynamics  of  cost  overrun  and  time  delay.  Although 
a  holistic  view  is  advocated,  introduced,  and  discussed  in  this  article, 
only  limited  aspects  are  ultimately  quantified.  Indeed,  only  a  series  of 
articles  would  do  justice  to  the  quantification  of  risks  associated  with  all 
factors  that  embody  software  nontechnical  risk.  By  their  nature,  quanti¬ 
fication  and  management  of  software  nontechnical  risk  (and  to  a  large 
extent  software  technical  risk)  involve  the  customer  and  the  shadow 
client  (the  U.S.  Congress,  in  the  case  of  DoD)  and  the  contractor(s). 
Organizational  interface  between  the  customer  and  the  contractor(s), 
the  state  of  technology  and  know-how,  the  complexity  of  the  specifica¬ 
tion  requirements,  add-on  modifications  and  refinements,  the  availabil¬ 
ity  of  appropriate  resources,  and  the  models  used  for  project  cost  esti¬ 
mation  and  schedule  projections  are  major  considerations. 

Since  each  element  is  in  itself  a  complex  entity  with  diverse  dimen¬ 
sions,  it  is  essential  to  recognize  which  characteristics  of  each  compo¬ 
nent  contribute  to  software  nontechnical  risk.  Only  by  understanding 
the  sources  of  risk  can  it  ultimately  be  prevented  and  managed. 

The  Customer 

The  term  customer  is  a  misnomer  because  it  connotes  a  singular  entity. 
Yet,  in  most  large-scale  software  engineering  systems,  such  as  DoD  sys¬ 
tems,  projects  are  initiated,  advocated,  nourished,  and  supported  by 
multiple  constituencies  with  some  common,  but  often  different,  goals 
and  objectives.  Furthermore,  for  DoD  projects,  there  is  also  the  shadow 
customer — the  U.S.  Congress,  itself  influenced  by  various  lobbyists,  power 
brokers,  and  stakeholders.  The  existence  and  influence  of  this  multiplic¬ 
ity  of  clients  on  the  ultimate  resources  for  the  development  of  software 
engineering  constitute  a  critical  source  of  software  risk.  It  is  not  uncom¬ 
mon  for  a  pressure  group  to  affect  either  the  design  specifications  and/ 
or  the  resources  allocated  for  a  specific  DoD  project,  and  thus  to  have 
an  impact  on  its  final  cost  and  its  completion  time.  The  “organizational 
maturity”  level  of  the  client  is  another  factor  that  influences  software 
nontechnical  risk.  A  client  that  possesses  internal  capabilities  to  com¬ 
municate  with  the  contractor(s)  on  both  the  technical  and  nontechnical 
levels  is  more  likely  to  have  a  better  understanding  and  therefore  man¬ 
agement  of  software  nontechnical  risk.  This  attribute  will  become  more 
evident  in  this  article  as  specific  quantitative  information  on  the  vari¬ 
ances  of  cost  and  schedule  is  solicited  in  the  proposed  methodological 
framework. 
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The  Contractor(s) 

Elaborate  procedures  and  protocols  describing  contractor  selection  for 
the  development  of  software  engineering  have  been  designed  and  are 
being  employed  by  government  agencies  and  corporations.  A  commonly 
accepted  axiomatic  premise  is  that  the  organizational  maturity  level  of 
the  contractor  and  the  experience,  expertise,  and  qualifications  of  its 
staff  markedly  affect  the  management  of  software  technical  and  non¬ 
technical  risks. 

The  Interface  Between  the  Customer  and  the  Contractor(s) 

One  of  the  dominant  factors  in  initiating  both  technical  and  nontechni¬ 
cal  risks  can  be  traced  to  the  organizational  interface  between  the  cus¬ 
tomer  and  the  contractors ).  Adequate  and  appropriate  communication 
between  the  two  parties,  and  the  understanding  and  the  appreciation  of 
each  other’s  role  throughout  the  life  cycle  of  the  software  development 
process  are  imperatives  to  the  prevention  and/or  control  of  potential  risks. 

The  State  of  Technology  and  Know-how 

Although  many  consider  the  contractor’s  access  to  knowledge  and  the 
access  to  appropriate  technology  to  be  major  factors  in  controlling  soft¬ 
ware  technical  risk,  these  factors  also  impact  software  nontechnical  risk. 
In  particular,  the  lack  of  access  to  appropriate  technology  or  a  deficient 
know-how  in  the  contractor’s  staff  is  likely  to  cause  a  measurable  time 
delay  in  the  completion  of  a  project  and  is  also  likely  to  cause  cost 
overrun. 

The  Complexity  of  the  Specification  Requirements 

The  more  unprecedented  the  client’s  project  specifications  in  terms  of 
advanced  and  emerging  technology,  the  higher  the  risk  of  time  delay  in 
its  completion  and  of  its  cost  overrun.  Most  systems  developed  by  the 
DoD  are  advancing  the  state  of  the  art  in  some  field  of  technology,  e.g., 
software  development,  stealth,  propulsion,  and  satellites.  The  require¬ 
ments  in  these  fields  are  necessarily  complex  since  the  parameters  are 
constrained  by  the  task  and  are  frequently  subject  to  modifications  be¬ 
cause  of  changing  technology. 

The  Add-on  Modifications  and  Refinements 

Add-on  modifications  and  refinements  are  viewed  by  many  as  an  Achil¬ 
les’  heel  in  terms  of  software  nontechnical  risk.  Although  these  add-on 
modifications  are  often  associated  with  software  nontechnical  risk,  they 
also  constitute  a  critical  source  of  software  technical  risk.  This  is  be¬ 
cause  not  all  modifications  are  appropriately  related  to  and  checked 
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against  the  original  design  to  ensure  ultimate  compatibility  and  har¬ 
mony.  Very  large  and  complex  systems  are  difficult  to  manage.  Systems 
are  now  developed  by  multiple  companies,  each  having  its  own  area  of 
expertise,  and  changes  often  ripple  through  the  entire  system.  A  wide 
range  of  factors  may  cause  mid-course  modifications;  however,  three 
categories  of  causes  emerge  from  this  range: 

a.  Threat  or  Need  Change:  When  a  new  threat  is  projected  or  a  new 
need  is  contemplated. 

b.  Improved  New  Technology:  When  a  new  technology  provides  im¬ 
proved  performance  or  quality,  such  as  a  new  sensor. 

c.  Obsolete  Technology  Replacement:  When  the  pre-selected  tech¬ 
nology  becomes  obsolete  before  the  contract  has  even  begun  or 
been  completed. 

The  Availability  of  Appropriate  Resources 

One  open  secret  in  government  procurement  and  occasionally  in  the 
private  sector  is  the  level  of  pre-allocated  funds  for  a  specific  project. 
The  competitive  zeal  of  contractors  often  outweighs  the  technical  judg¬ 
ment  of  their  professional  staff;  the  outcome  is  a  bid  that  is  close  to  the 
pre-allocated  funds  by  the  client  even  though  it  is  clear  to  the  bidder 
that  the  job  with  its  specification  requirements  cannot  be  delivered  at 
that  level  of  funding.  This  not-uncommon  phenomenon  is  dramatically 
illustrated  in  numerous  documented  examples  by  Hedrick  Smith  (1988). 

The  standard  technique  is  to  get  a  project  started  by 
having  the  prime  contractor  give  a  low  initial  cost  esti¬ 
mate  to  make  it  seem  affordable  and  wait  to  add  fancy 
electronics  and  other  gadgets  much  later  through  engi¬ 
neering  “change  orders,”  which  jack  up  the  price  and  the 
profits.  Anyone  who  has  been  through  building  or  re¬ 
modeling  a  house  knows  the  problem.  “This  is  called  the 
buy-in  game,”  an  experienced  Senate  defense  staff  spe¬ 
cialist  confided. 

The  Models  Used  for  Project  Cost  Estimation  and  Schedule  Projection 

A  number  of  models  are  used  to  estimate  project  cost  and  its  comple¬ 
tion  schedule.  Constructive  Cost  Model  (COCOMO)  (Boehm  1981)  and 
Program  Evaluation  and  Review  Technique  (PERT)  are  representative 
examples.  Models  can  be  potent  tools  when  they  are  well  understood. 
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are  supported  by  an  appropriate  database,  and  adhere  to  the  assump¬ 
tions  upon  which  they  are  designed  to  operate.  The  complexities  of  such 
models,  however,  often  result  in  their  misuse  and/or  invalid  interpreta¬ 
tions  of  their  results.  They  thereby  ironically  become  a  source  of  soft¬ 
ware  nontechnical  risk.  The  successful  application  of  the  proposed  meth¬ 
odological  framework,  however,  does  not  depend  on  the  specific  model 
used  by  either  the  contractor  or  the  customer  to  estimate  either  the  cost 
or  the  schedule. 

From  the  above  it  seems  that  the  sources  that  contribute  to  software 
nontechnical  risk  are  organizational  and  technical  in  nature;  they  stem 
from  failures  associated  with  the  contractor  as  well  as  the  customer.  In 
terms  of  the  contractor,  these  failures  primarily  originate  from  and  are 
functions  of  such  elements  as: 

a.  the  organizational  maturity  level; 

b.  the  process  and  procedures  followed  in  the  assessment  of  the 
project’s  cost  and  schedule; 

c.  the  level  of  honesty  exhibited  by  management  in  communicating  the 
real  cost  and  schedule  to  the  customer  (and  of  course  vice  versa); 

d.  the  extent  and  level  of  new  and  unprecedented  technology  im¬ 
posed  on  the  project; 

e.  the  level  of  experience  and  expertise  of  the  staff  engineers  in  soft¬ 
ware  engineering  in  general  and  in  the  application  domain  in  par¬ 
ticular; 

f.  the  level  of  experience  and  expertise  of  the  management  team  in 
software  engineering; 

g.  the  overall  competence  of  the  team  developing  the  software; 

h.  financial  and  competitive  considerations; 

i.  immature  technology,  methods,  and  tools; 

j.  the  use  of  technology  in  new  domains; 

k.  new  combinations  of  methods  and  tools  in  new  ways  and  their  use 
in  a  new  software  development  environment; 
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1.  requirement  modifications  causing  changes  in  the  system’s  archi¬ 
tecture. 

In  terms  of  the  customer,  the  natures  of  organizational  failures  par¬ 
tially  overlap  those  of  the  contractor’s,  but  also  have  distinctive  charac¬ 
teristics: 

a.  the  process  and  procedures  followed  in  the  assessment  of  the  project 
cost  and  schedule; 

b.  the  level  of  specificity  at  which  the  system  and  software  require¬ 
ments  are  detailed; 

c.  the  number  of  changes  and  modifications  requested  by  the  cus¬ 
tomer  during  the  software  development  process  (changes  which 
generally  introduce  many  new  errors)  are  often  not  harmonious 
with  earlier  specification  requirements; 

d.  the  commitment  of  project  management  (associated  with  the 
customer’s  organization)  to  closely  monitor  and  oversee  the  soft¬ 
ware  development  process; 

e.  the  specific  requirements  for  technology,  e.g.,  specific  compiler, 
data  base  management  systems; 

f.  the  level  of  honesty  exhibited  by  management  in  communicating 
the  real  cost  to  the  “real  client”  (e.g.,  the  Department  of  Defense 
as  a  client  and  the  U.S.  Congress  as  the  “real  client”). 

BASIS  FOR  VARIANCES  IN  COST  ESTIMATION 

Most,  if  not  all,  developers  of  large  complex  software  systems  use  cost 
models  to  estimate  their  costs.  These  models  are  based  on  a  set  of 
relationships  based  on  such  parameters  as  the  size  and  complexity  of  the 
software,  the  experience  level  of  the  software  developer,  and  the  type  of 
application  within  which  the  software  will  be  used.  Different  models 
generate  different  weights  or  levels  of  importance  for  these  parameters, 
and  not  all  models  use  the  same  parameters.  Therefore,  one  cost  model 
can  lead  to  a  radically  different  cost  estimate  than  another  just  on  the 
basis  of  which  parameters  are  used  in  the  model  and  how  they  are 
implemented.  Even  if  the  parameters  are  used  consistently,  however, 
different  developers  will  probably  not  agree  on  the  value  or  weight  of 
the  parameter  in  the  first  place.  Many  organizations,  in  fact,  consider 
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their  interpretations  of  these  parameters  to  contribute  to  their  “com¬ 
petitive  edge”  because  the  definition  affects  their  ability  to  determine 
costs  accurately.  For  example,  an  organization  that  has  experience  in 
developing  space  system  software  may  not  have  the  same  perception  of 
difficulty  when  developing  a  complex  avionics  software  system  as  would 
an  organization  that  has  significant  experience  in  that  area.  Their  under¬ 
standing  of  space  systems,  however,  will  alter  their  definition  of  the 
avionics  system  parameters.  Do  developers  with  little  experience  overes¬ 
timate  or  underestimate  the  complexity  of  the  task  because  of  how  they 
define  these  parameters?  As  stated  in  the  beginning,  these  are  questions 
central  to  this  paper:  What  are  the  sources  of  risk  associated  with  project 
cost  estimation?  How  can  such  risk  be  quantified? 

Although  creating,  maintaining,  and  updating  project  cost  estimation 
metrics  and  parameters  are  extremely  important  for  an  organization,  it 
is  nevertheless  unlikely  that  a  future  project  will  be  similar  enough  to 
previous  projects  to  merit  directly  importing  these  metrics  or  param¬ 
eters;  such  metrics  and  parameters  may  not  be  directly  applicable  with¬ 
out  appropriate  modifications.  Indeed,  cost  estimators  are  required  to 
(and  do)  use  judgment  when  applying  these  parameters  to  a  new  project 
requirement.  Furthermore,  cost  estimation  constitutes  a  critical  area 
with  regard  to  the  sources  of  risk  for  software  development,  which  is 
without  parallel  to  other  fields.  For  example,  if  a  contractor  was  estimat¬ 
ing  the  cost  to  construct  a  building  with  50  stories,  yet  had  previously 
only  built  structures  with  a  maximum  of  10  stories,  the  contractor  would 
not  just  increase  the  estimate  five-fold.  The  contractor  would  question 
the  basic  foundations  and  relevance  of  extending  the  10-story  model  to 
the  new  structure  parameters.  However,  in  software,  it  is  not  uncommon 
to  increase  estimates  for  new  projects  by  a  factor  of  five  from  previous 
projects  of  one-fifth  the  size  and  complexity.  Many  new  systems  have 
size  estimates  of  over  1,000,000  lines  of  code  even  though  the  develop¬ 
ers  have  little  experience  with  systems  of  this  size. 

Another  example  is  in  the  use  of  commercial  off-the-shelf  (COTS) 
software.  The  original  assumption  that  a  commercial  database  manage¬ 
ment  system  (DBMS)  can  be  used  to  meet  customer  requirements  may 
change  if  the  customer  requires  features  not  supported  by  DBMS  sup¬ 
pliers.  Such  changes  may  have  serious  ramifications  for  the  cost  estimate 
depending  on  how  the  developer  plans  to  solve  the  problem.  If  the 
developer  chooses  to  subcontract  out  the  effort  and  deal  with  the  sub¬ 
contractor  in  a  way  similar  to  dealing  with  the  DBMS  vendor,  this  has 
ramifications  for  the  risk  associated  with  the  subcontractor — an  impor¬ 
tant  subject  that  will  be  discussed  later.  The  alternative  is  for  the  devel¬ 
oper  to  undertake  the  development  of  his  or  her  own  DBMS.  This 
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ances  or  the  corresponding  PDFs.  In  the  latter  case,  the  contractor 
may  choose  to  use  a  triangular  distribution,  the  fractile  method,  or 
any  other  means  that  the  contractor  believes  will  provide  the  most 
accurate  estimate.  Clearly,  the  contractor  can  and  is  likely  to  use 
models  and  other  tools  to  generate  the  required  PDFs.  At  the  same 
time  the  customer’s  staff  will  generate  its  own  PDFs  for  cost  and 
completion  time.  The  customer  is  now  able  to  compare  not  only  the 
expected  values  (the  means)  of  each  contractor’s  cost  and  comple¬ 
tion  time,  but  also  the  variances  of  these  estimates.  Furthermore,  a 
comparison  of  the  extremes  of  each  PDF  provides  valuable  informa¬ 
tion  to  the  customer  about  each  contractor’s  capabilities  and  compat¬ 
ibility.  Although  some  of  the  efficacious  attributes  of  the  method¬ 
ological  framework  will  be  better  understood  after  we  introduce  the 
section  on  “Risk  of  Extreme  Events”  and  the  example  problems,  an 
overview  of  the  required  steps  may  clarify  the  process: 

1.  Use  the  fractile  method,  the  triangular  distribution,  or  any  other 
approach  to  quantify  the  variances  associated  with  project  cost 
estimates  and  the  completion  schedule. 

2.  Assess  the  contractor’s  capability  to  deliver  the  product  and  to 
estimate  the  likely  variance  of  the  project’s  cost  and  schedule 
through  SEI’s  Software  Risk  Taxonomy-Based  Questionnaire.  The 
SEI  taxonomy  and  the  accompanying  questionnaire  provide  a 
framework  for  identifying  the  technical  uncertainties  in  a  project 
and  the  root  causes  of  these  uncertainties.  It  also  provides  a  method 
for  assessing  the  honesty  and  credibility  of  the  contractor’s  analy¬ 
sis  and  figures. 

3.  Evaluate,  in  a  quantitative  way,  any  discrepancy  between  the  variance 
assessments  of  the  contractor  and  the  customer.  (The  quantification 
is  likely  to  lead  to  significant  information  about  the  likelihood  of 
extreme  events  and  their  potential  consequences  for  the  entire  project.) 

4.  Investigate  and  understand  the  contractor’s  assumptions  in  estimating 
variances.  This  information  will  enable  the  customer’s  staff  to  take 
appropriate  measures  to  mitigate  software  nontechnical  risk. 

5.  Integrate  the  information  on  the  contractor  derived  from  (a)  the 
quantitative  variances  received  on  the  projected  cost  and  comple¬ 
tion  schedule  with  (b)  the  results  generated  from  the  Software 
Risk  Taxonomy-Based  Questionnaire. 
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Project  Cost  Increase  (%) 


Figure  4.  Probability  Density  Function  for  Project  Cost  Increase 

for  Contractor  A 


stitutes  the  basis  for  the  generation  of  probability  density  functions  (PDFs) 
associated  with  the  project's  cost  and  its  completion  time.  Assuming  that 
the  customer  (with  the  assistance  of  a  MITRE-type  organization,  if 
needed)  is  also  capable  of  providing  (for  comparative  purposes  with  the 
clients’  variances)  basic  variance  information,  which  allows  the  genera¬ 
tion  of  the  customer’s  PDFs  for  the  projected  cost  and  completion  time, 
then  the  following  method  will  be  useful.  A  mechanism  is  developed 
here  that  enables  the  customer  to  compare  various  contractors’  probabi¬ 
listic  estimates  of  several  attributes  and  characteristics  to  its  own  esti¬ 
mate.  To  use  either  of  the  two  approaches  to  estimate  cost  or  schedule 
variances,  the  contractor  should  familiarize  the  team  making  such  esti¬ 
mation  with  the  intricacy  of  these  approaches  and  alert  it  to  the  cogni¬ 
tive  biases  inherent  in  such  an  estimation  process.  In  their  quest  to 
quantify  these  human  biases,  Alpert  &  Raiffa  (1982)  conducted  several 
experiments  over  two  decades  ago  and  arrived  at  the  conclusion  that 
with  appropriate  training,  the  use  of  the  fractile  method  can  be  very 
effective.  The  question  of  gaming  and  the  manipulation  of  the  approach 
by  some  contractors  to  gain  advantage  is  discussed  under  the  subhead 
“Comparative  Analysis  Among  Contractors.” 

The  proposed  methodological  framework  requires  a  number  of 
steps.  First,  the  customer  requires  that  each  bidding  contractor  sub¬ 
mit  either  basic  information  on  the  cost  and  completion  time  vari- 
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ances  or  the  corresponding  PDFs.  In  the  latter  case,  the  contractor 
may  choose  to  use  a  triangular  distribution,  the  fractile  method,  or 
any  other  means  that  the  contractor  believes  will  provide  the  most 
accurate  estimate.  Clearly,  the  contractor  can  and  is  likely  to  use 
models  and  other  tools  to  generate  the  required  PDFs.  At  the  same 
time  the  customer’s  staff  will  generate  its  own  PDFs  for  cost  and 
completion  time.  The  customer  is  now  able  to  compare  not  only  the 
expected  values  (the  means)  of  each  contractor’s  cost  and  comple¬ 
tion  time,  but  also  the  variances  of  these  estimates.  Furthermore,  a 
comparison  of  the  extremes  of  each  PDF  provides  valuable  informa¬ 
tion  to  the  customer  about  each  contractor’s  capabilities  and  compat¬ 
ibility.  Although  some  of  the  efficacious  attributes  of  the  method¬ 
ological  framework  will  be  better  understood  after  we  introduce  the 
section  on  “Risk  of  Extreme  Events”  and  the  example  problems,  an 
overview  of  the  required  steps  may  clarify  the  process: 

1.  Use  the  fractile  method,  the  triangular  distribution,  or  any  other 
approach  to  quantify  the  variances  associated  with  project  cost 
estimates  and  the  completion  schedule. 

2.  Assess  the  contractor’s  capability  to  deliver  the  product  and  to 
estimate  the  likely  variance  of  the  project’s  cost  and  schedule 
through  SEI’s  Software  Risk  Taxonomy-Based  Questionnaire.  The 
SEI  taxonomy  and  the  accompanying  questionnaire  provide  a 
framework  for  identifying  the  technical  uncertainties  in  a  project 
and  the  root  causes  of  these  uncertainties.  It  also  provides  a  method 
for  assessing  the  honesty  and  credibility  of  the  contractor’s  analy¬ 
sis  and  figures. 

3.  Evaluate,  in  a  quantitative  way,  any  discrepancy  between  the  variance 
assessments  of  the  contractor  and  the  customer.  (The  quantification 
is  likely  to  lead  to  significant  information  about  the  likelihood  of 
extreme  events  and  their  potential  consequences  for  the  entire  project.) 

4.  Investigate  and  understand  the  contractor’s  assumptions  in  estimating 
variances.  This  information  will  enable  the  customer’s  staff  to  take 
appropriate  measures  to  mitigate  software  nontechnical  risk. 

5.  Integrate  the  information  on  the  contractor  derived  from  (a)  the 
quantitative  variances  received  on  the  projected  cost  and  comple¬ 
tion  schedule  with  (b)  the  results  generated  from  the  Software 
Risk  Taxonomy-Based  Questionnaire. 
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6.  Use  the  risk  ranking  and  filtering  (RRF)  method  to  rank  the  risks 
associated  with  each  prospective  contracting  organization;  then, 
compare  those  risks  against  one  another  and  against  an  estab¬ 
lished  norm. 

Although  these  methods  and  processes  may  not  provide  an  optimal 
approach  for  selecting  the  best  or  most  valid  estimate,  they  do  provide  a 
foundation  that  is  systematic  and  repeatable  to  allow  the  evaluators  to 
gain  significant  knowledge  and  insight  into  the  estimators’  assumptions. 
This  insight  is  critical  from  two  perspectives;  it  enables  the  customer  to: 

•  evaluate  whether  the  contractors’  estimates  and  assumptions  are 
valid  and  consistent  with  the  specifications;  and 

•  establish  a  foundation  by  which  to  evaluate  and  judge  future  changes 
to  cost  and  schedule  estimates. 

These  methods  and  processes  also  provide  a  mechanism  for  the 
customer’s  evaluation  team  to  document  the  assumptions  and  risks  in  a 
cost  or  schedule  estimate,  identify  the  root  issues  associated  with  these 
assumptions  and  risks,  and  organize  this  information  within  the  tax¬ 
onomy  framework.  This  information  can  then  be  used  to  measure 
progress  and  can  also  be  used  as  a  metric  against  future  cost  and  sched¬ 
ule  estimates. 

RISK  OF  EXTREME  EVENTS 

In  general,  the  estimates  of  the  most  likely  project  cost  provided  by  the 
dominant  number  of  contractors  will  be  within  a  close  range  of  one 
another.  Since  assessing  and  ultimately  preventing  potential  major  cost 
overruns  and  time  delays  are  of  major  concern  in  this  paper,  our  interest 
here  is  in  what  can  go  wrong  in  the  extremes,  i.e.,  in  the  behavior  of  the 
tail  of  the  distribution.  This  can  best  be  captured  through  the  condi¬ 
tional  expected  value  of  extreme  events.  The  conditional  expected  value 
of  risk,  denoted  by  f4(«)  (which  will  be  defined  later),  can  provide  valu¬ 
able  information  that  supplements  and  complements  the  average  cost  or 
most  likely  cost  estimates. 

Most  analysts,  who  use  probabilistic  quantitative  methods  to  measure 
the  risk  of  project  cost  overruns  and  delays  in  the  completion  schedule, 
resort  to  the  most  common  mathematical  construct  for  the  quantifica¬ 
tion  of  risk — the  expected  value  of  risk.  Whether  the  probabilities  asso¬ 
ciated  with  the  universe  of  events  are  viewed  by  the  analyst  as  discrete 
or  continuous,  the  expected  value  of  risk  is  an  operation  that  essentially 
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multiplies  each  event  by  its  probability  of  occurrence  and  sums  (or  inte¬ 
grates)  all  these  products  over  the  entire  universe  of  events.  This  opera¬ 
tion  literally  commensurates  adverse  events  of  high  consequences  and 
low  probabilities  of  exceedance  with  events  of  low  consequences  and 
high  probabilities  of  exceedance.  Indeed,  the  expected  value  masks  the 
extremes  and  hides  the  effects  of  less  likely  outcomes. 

The  misuse,  misinterpretation,  and  fallacy  of  the  expected  value,  when 
it  is  used  as  the  sole  criterion  for  risk  in  decision  making,  are  discussed 
elsewhere  (Haimes  &  Chittister  1993,  Asbeck  &  Haimes  1984).  Many 
experts  are  becoming  convinced  of  the  grave  limitations  of  the  tradi¬ 
tional  and  commonly  used  expected-value  concept  and  so  are  augment¬ 
ing  the  expected  value  of  risk  with  a  supplementary  measure — the  con¬ 
ditional  expectation — by  which  decisions  about  extreme  and  catastrophic 
events  are  not  averaged  out  with  more  commonly  occurring  high-fre- 
quency/low-consequence  events. 

The  partitioned  multiobjective  risk  method  (PMRM)  is  a  risk  analysis 
method  developed  for  solving  probabilistic  multiobjective  problems  with 
a  focus  on  extreme  events  (Asbeck  &  Haimes,  1984).  Instead  of  using 
the  traditional  expected  value  of  risk,  the  PMRM  generates  a  number  of 
conditional  expected-value  functions,  known  as  risk  functions,  which  rep¬ 
resent  the  risk  given  that  the  damage  falls  witin  specific  ranges  of  the 
probability  of  exceedance  or  within  a  range  of  adverse  consequences 
(generically  termed  as  damages).  Before  the  PMRM  was  developed, 
problems  with  at  least  one  random  variable  were  solved  by  computing 
and  minimizing  the  unconditional  expectation  of  the  random  variable 
representing  the  specific  damage.  In  contrast,  the  PMRM  isolates  a 
number  of  damage  ranges  (by  specifying  partitioning  probabilities)  and 
generates  conditional  expectations  of  damage  given  that  the  damage 
falls  within  a  particular  range.  In  this  manner,  the  PMRM  can  generate 
a  number  of  risk  functions,  one  for  each  range,  which  are  then  aug¬ 
mented  with  the  original  optimization  problem  as  new  objective  func¬ 
tions.  In  this  paper  the  discussion  will  be  limited  to  one  conditional 
expected  value  of  extreme  events,  denoted  by  f4(«). 

The  conditional  expectations  of  a  problem  are  found  by  partitioning 
the  problem’s  probability  axis  and  mapping  these  partitions  onto  the 
damage  axis.  The  damage  axis  in  this  case  can  be  project  cost  overrun  in 
terms  of  dollars  or  percentage  of  overage  above  the  contracted  level; 
alternatively,  the  damage  can  represent  time  delay  in  terms  of  months 
or  weeks  or  in  terms  of  percentages  in  relation  to  the  original  time 
schedule.  Consequently,  the  damage  axis  is  partitioned  into  correspond¬ 
ing  ranges.  A  conditional  expectation  is  defined  as  the  expected  value  of  a 
random  variable  given  that  this  value  lies  within  some  prespecified  prob- 
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ability  range  (or  within  some  prespecified  damage  range).  Clearly,  the  val¬ 
ues  of  conditional  expectations  are  dependent  on  where  the  probability 
axis  (or  the  damage  axis)  is  partitioned.  The  choice  of  where  to  partition 
is  made  subjectively  by  the  analyst  in  response  to  the  extreme  character¬ 
istics  of  the  decision-making  problem. 

A  continuous  random  variable  X  of  damages  (e.g.,  cost  overrun  or 
time  delay)  has  a  cumulative  distribution  function  (CDF)  P(x),  and  a 
probability  density  function  (PDF)  p(x),  which  are  defined  by  the  rela¬ 
tionships 

CDF:  P(x)  =  prob[X  <  x]  (1) 


PDF:  p(x)  = 


dP  (x) 
— d)T 


and 


(2) 


The  CDF  represents  the  nonexceedance  probability  of  x.  The 
exceedance  probability  of  x  is  defined  as  the  probability  that  X  is  ob¬ 
served  to  be  greater  than  x  and  is  equal  to  one  minus  the  CDF  evaluated 
at  x. 

The  expected  value,  average,  or  mean  value  of  the  random  variable  X 
is  defined  as 

E[X]  =  jp°(x)  dx  (3) 

0 


For  the  purpose  of  this  article,  a  modified  version  of  the  PMRM  is 
presented  to  simplify  the  mathematical  discussion  and  to  focus  the  analysis 
on  the  conditional  risk  of  extreme  events.  Let  1  -  a,  where  0  <  a  <  1, 
denote  an  exceedance  probability  that  partitions  the  domain  of  X  into 
two  ranges.  On  a  plot  of  exceedance  probability,  there  is  a  unique  dam¬ 
age  b  on  the  damage  axis  that  corresponds  to  the  exceedance  probability 
1  -  a  on  the  probability  axis.  Damages  (e.g.,  cost  overruns  or  time  delays 
in  project  completion  schedule)  less  than  b  are  considered  to  be  of  low 
to  moderate  severity;  damages  greater  than  b  are  of  high  severity.  The 
partitioning  of  risk  into  two  severity  ranges  is  illustrated  in  Figure  2.  If 
the  partitioning  probability  a  is  specified,  for  example,  to  be  0.95,  then  b 
is  the  95th  percentile. 

The  conditional  expected  damage  (given  that  the  damage  is  within 
that  particular  range)  provides  a  measure  of  the  risk  associated  with  the 
range.  The  measure  of  conditional  expected  value  of  risk  of  interest 
here  is  that  of  low  exceedance  probability  and  high  severity,  denoted  by 
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f4(»).  High  severity  may  mean  a  high  cost  overrun  or  a  high  time  delay  in 
the  project’s  scheduled  completion.  The  function  f4(»)  is  the  expected 
value  of  X,  given  that  x  is  greater  than  or  equal  to  b: 

f4(.)  =  E[X  |  x  >  b]  (4) 

For  any  probability  of  exceedance,  one  can  generate  the  traditional, 
unconditional  (common)  expected  value  of  risk  (of  cost  overrun  and/or 
of  time  delay)  denoted  by  f5(»),  and  the  conditional  expected  value  of 
risk  of  extreme  events  (of  same)  denoted  by  f4(*).  Note  that 

.  oo 

J  x  p(x)  dx 

f4(.)  =  (5) 

Jp(x)  dx 

b 

f'5C)  =  Jp(x)dx  (6) 

0 

where  p(x)  is  the  probability  density  function. 

EXAMPLE  PROBLEM 

DoD  is  considering  the  introduction  of  a  new  strategic  airplane  that  will 
constitute  the  flagship  of  the  Air  Force  as  we  enter  the  third  millen¬ 
nium.  Aware  of  the  powershift  from  hardware  to  software  in  technology 
and  the  emerging  centrality  of  software  as  the  overall  system  integrator 
and  coordinator,  DoD  considers  the  development  of  software  for  this 
airplane  to  be  of  paramount  importance  (Chittister  &  Haimes  1994). 
The  Air  Force  commissions  the  assistance  of  a  support  organization  to 
develop,  in  collaboration  with  its  own  staff,  specifications  and  a  request 
for  proposal  (RFP)  for  designing,  prototyping,  and  developing  the  soft¬ 
ware  needed  for  the  flagship  airplane.  Following  a  detailed  and  tedious 
process  of  qualifying  prospective  bidders,  the  Air  Force  issues  an  RFP 
for  the  development  of  the  required  software  engineering.  This  time, 
however,  the  RFP  includes  items  that  had  not  been  requested  previ¬ 
ously.  For  example,  the  RFP  requires  that  each  contractor  provide  vari¬ 
ances  along  with  the  estimated  project’s  cost  and  completion  schedule, 
instead  of  the  commonly-practiced  requirement  of  single  deterministic 
values.  The  RFP  leaves  it  up  to  the  contractors  to  determine  the  form 
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that  these  variances  take,  including,  if  the  contractor  so  desires,  the  type 
of  PDF  selected  for  each  estimate.  The  Air  Force  and  its  support  team, 
planning  to  use  the  same  approach  themselves  in  evaluating  the  various 
proposals,  recommends  in  the  RFP  the  optional  use  of  the  fractile 
method  or  the  triangular  distribution  when  complete  statistical  informa¬ 
tion  is  not  readily  available. 

To  capture  the  mathematical  details  entailed  in  the  process  of  devel¬ 
oping  representative  probability  density  functions  (PDFs)  for  cost  and 
completion  time,  a  step-by-step  procedure  using  the  fractile  method 
(adopted  by  Contractors  A  and  B)  is  presented  here.  For  pedagogical 
purposes,  a  detailed  analysis  is  presented  for  Contractor  A  only.  Similar 
analysis  should  be  followed  for  Contractor  B  and  the  customer.  The 
team  from  Contractor  A  estimates  a  most  likely  cost  of  $150  million.  After 
considerable  brainstorming  sessions,  the  following  information  emerges: 

•  Best  case  project  cost  increase  =  0%  (i.e.,  project  cost  is  $150 
million); 

•  Worst  case  project  cost  increase  =  50%  (i.e.,  project  cost  increase 
is  $75  million,  for  a  total  of  $225  million);; 

•  Median  value  of  project  cost  increase  (equal  likelihood  of  being 
greater  or  less  than  this  value)  =  15%  (i.e.,  project  cost  increase  is 
$22.5  million,  for  a  total  of  $172.5  million); 

•  50-50  chance  that  the  actual  project  cost  would  be  within  5%  of  the 
15%  median  estimate  (i.e.,  project  cost  increase  is  (15  ±  5)%). 

From  the  above  information,  the  following  fractiles  (percentiles)  are 
readily  determined. 

•  The  best  scenario  of  no  cost  overrun  (0%  cost  increase,  i.e.,  a  total 
cost  of  $150  million)  represents  the  0.00  fractile  (0  percentile); 

•  The  worst  scenario  of  50%  cost  overrun  (a  total  cost  of  $225  mil¬ 
lion)  represents  the  1.00  fractile  (100th  percentile); 

•  The  median  value  of  15%  cost  overrun  (a  total  cost  of  $172.5  mil¬ 
lion)  represents  the  0.50  fractile  (50th  percentile); 

•  The  0.25  fractile  (25th  percentile)  is  (15-5)%  =  10%  increase  over 
$150  million  (a  total  cost  of  $165  million). 
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•  The  0.75  fractile  (75th  percentile)  is  (15+5)%  =  20%  increase  over 
$150  million  (a  total  cost  of  $180  million). 

The  above  assessment  of  project  cost  for  contractor  A  and  similarly 
for  Contractor  B  and  for  the  customer  is  summarized  in  Table  1  and  is 
used  as  a  basis  for  constructing  the  corresponding  cumulative  distribu¬ 
tion  function  (CDF)  for  Contractor  A.  (See  Figure  3.) 

The  CDF  (Figure  3)  can  now  be  represented  in  terms  of  a  PDF 
(Figure  4).  To  construct  the  PDF,  one  must  be  guided  by  the  following 
principles: 

(a)  The  area  under  the  shaded  area  (the  PDF)  must  be  equal  to  1. 

(b)  The  first  quartile  in  Figure  3  (representing  25%  of  the  probabili¬ 
ties)  spans  a  cost  overrun  from  0%  to  10%.  Thus,  the  correspond¬ 
ing  area  of  the  PDF  (Figure  4)  must  be  equal  to  one-fourth  of  the 
total  area,  i.e.,  0.25.  Dividing  0.25  by  10  yields  a  height  of  0.025  for 
the  first  rectangle  in  Figure  4.  Similarly  for  the  other  three  quartiles, 
each  of  the  second  and  third  quartiles  spans  5%  of  project  cost 
increase.  Thus,  the  height  of  the  rectangle  of  the  PDF  (Figure  4) 
is  0.25  on  the  probability  axis  and  when  divided  by  5  yields  a 
height  of  0.05  on  the  probability  axis.  Finally,  the  last  quartile 
spans  a  cost  overrun  of  30%  (from  20%  to  50%).  Thus,  the 
height  of  the  rectangle  (on  the  probability  axis)  is  0.25  divided 
by  30  which  yields  a  height  of  0.008.  Figure  5  depicts  the 
exceedance  probability  (1-CDF)  vs.  project  cost  increase.  To  fo¬ 
cus  on  the  exceedance  probability  of  a  major  cost  overrun,  say 
between  20%  and  50%,  only  that  part  of  Figure  5  is  depicted  in 
Figure  6.  Note  that  by  just  using  basic  rules  from  geometry,  one 
can  relate  the  exceedance  probability  to  any  project  cost  increase 
x,  for  20%  _  x  50%. 

The  expected  value  of  the  percentage  of  project  cost  increase,  f5(«), 
can  be  determined  from  geometry  (Figure  4): 

(10  -  0)  (15  -  10)  (20  - 15) 

f5(»)  =  0.25  [  0  +  — 2  ]  +  0.25  t10  +  — 2 — 1  +  °-25  t15  +  — 2 — J 

(50  -  20) 

+  0.25  [20  +  - — 2 — 1 

=  0.25  (5)  +  0.25  (12.5)  +  0.25  (17.5)  +  0.25  (35) 

=  0.25  (70)  =  17.5%  (i.e.,  total  cost  of  $(150  +  26.25)  million) 
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The  expected  value  of  the  percentage  of  project  cost  increase  may 
also  be  calculated  using  Equation  3.  Note  that  the  expected  value  of  cost 
overrun  of  $26.25  million  (i.e.,  total  cost  of  $176.25  million)  for  Contrac¬ 
tor  A  does  not  provide  any  vital  information  on  the  probable  extreme 
behavior  of  project  cost.  Also  note  that  there  is  a  one-to-one  functional 
relationship  between  the  probability  axis  and  the  percentage  of  project 
cost  increase  as  is  depicted  in  Figure  5.  For  example,  there  is  0.1  prob¬ 
ability  (one  chance  in  10)  that  project  cost  increase  will  be  equal  to  or 
above  38%.  This  result  is  generated  as  follows  (here  we  are  interested  in 
the  probability  of  exceedance  of  0.1,  i.e.,  a  =  0.90,  or  (1-a)  =  0.10): 

x  -  20  ab  0.25 -(1-a) 

50-20  ~~  ac  “  0.25 

30(1  -a) 

Thus,  x  —  30  -  q  25  +  20  —  38%  for  a  =  0.9 

Alternatively,  we  can  compute  from  Figure  6  the  partition  point  x 
(the  percentage  of  increase  in  cost)  that  corresponds  to  a  probability  of 
0.1  as  shown  below: 

(1  -  a)  =  (50  -  x)  h 

where  h  is  the  height  in  the  probability  axis 
0.25 

h  _  50-20  —  0.0083 

x  —  50  -  (-^)  =  50  -  =  38%  for  a  =  0.9 

The  conditional  expected  value  of  project  cost  can  be  calculated  for  a 
couple  of  scenarios  to  shed  light  on  the  behavior  of  the  tail  of  the  PDF. 
For  example,  given  (from  Figures  5  and  6)  that  there  is  0.1  probability 
of  project  cost  overrun  that  would  be  equal  to  or  exceed  38%  of  its 
original  scheduled  budget,  management  might  be  interested  in  answer¬ 
ing  the  following  question:  What  is  the  conditional  expected  value  of 
extreme  cost  overrun  beyond  the  38%  (or  extreme  cost  overrun  with 
exceedance  probability  that  is  below  0.1)?  Or  posed  differently,  within 
the  range  of  exceedance  probabilities  between  0.1  and  0.0  and  range  of 
cost  overruns  between  38%  and  50%,  what  is  the  expected  value  of 
project  cost  overrun?  Note  that  (a)  the  maximum  cost  overrun  was  pre¬ 
dicted  not  to  exceed  50%;  (b)  the  conditional  expected  value  is  the 
common  expected  value  limited  between  specific  levels  of  cost  overruns 
instead  of  the  entire  range  of  possible  cost  overruns;  and  (c)  the  ex- 
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pected  value  is  a  weighted  average  of  possible  cost  overruns  multiplied 
by  their  corresponding  probabilities  of  occurence  and  summed  over  that 
entire  range. 

Using  Equation  3,  the  common,  unconditional  expected  value  of  cost 
overrun,  f5(*),  was  calculated  earlier  to  be  17.5%.  Similarly,  the  condi¬ 
tional  expected  value  of  cost  overrun  under  the  scenario  of  0.1  probabil¬ 
ity  of  exceeding  the  original  cost  estimate  (by  38%  or  by  $57  million) 
computed  using  equation  5  yields  f4(»)  =  44%.  Note  that  the  PDF  of 
cost  overrun  portion  from  20%  and  beyond  is  a  linear  function.  Alterna¬ 
tively,  the  conditional  expected  value  can  be  computed  as  the  mean  of 
the  shaded  area  in  Figure  7. 

f4(.)  =  38  +  (-5-0'— )  =  44% 

In  other  words,  the  adjusted  (conditional)  expected  value  of  cost  over¬ 
run,  when  it  is  in  the  range  of  38%  to  50%  of  the  original  scheduled 
cost,  is  44%. 

Unless  the  project  is  a  cost-plus  contract,  the  interpretation  of  these 
results  should  alarm  top  management  of  Contractor  A;  although  the 
expected  cost  overrun  of  the  proposed  budget  is  17.50%  above  the  bud¬ 
geted  cost  of  $150  million,  there  is  a  10%  chance  (0.1  probability)  that 
the  cost  overrun  will  exceed  38%  of  the  budgeted  cost!  Furthermore,  at 


Figure  5.  Exceedance  Probability  For  Project  Cost  Increase 
for  Contractor  A 
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Figure  6.  Computing  the  Partition  Point  on  the  Damage  Axis 

for  Contractor  A 


Figure  7.  Computing  the  Conditional  Expected  Value  for  Contractor  A 
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a  10%  chance  of  cost  overrun,  the  conditional  expected  value  of  cost 
overrun  that  exceeds  38%  is  44%  above  the  original  budget,  i.e.,  an 
exceedance  of  $66  million;  in  other  words,  under  these  conditions,  the 
conditional  expected  value  of  the  total  cost  will  be  ($150  +  66)  million 
=  $216  million. 

It  is  worthwhile  to  clarify  at  this  point  the  meaning  of  the  two  distinct 
terms  of  cost  overrun:  38%  and  44%.  The  term  38%  cost  overrun  corre¬ 
sponds  to  a  single  probability  point  and  is  derived  directly  from  Figure  6. 
The  term  44%,  on  the  other  hand,  represents  the  conditional  expected 
value,  the  averaging  of  all  the  probabilities  from  0.10  to  zero  multiplied 
by  the  corresponding  cost  overruns  from  38%  to  50%,  summed  as  ap¬ 
propriate  and  scaled.  Thus 

f4(«)  =  E  [X  |  >  38%  cost  overrun]  =  44% 

or  equivalently, 

f4(»)  =  E  [X  |  >  $207  million]  =  $66  million 

It  is  constructive  to  further  clarify  the  information  summarized  in 
Table  2.  Consider  the  customer’s  column.  According  to  the  customer’s 
estimates  the  common,  unconditional  expected  value  of  cost  overrun  is 
11.25%.  Through  mathematical  calculations  on  the  basis  of  the  informa¬ 
tion  provided  by  the  customer  (as  shown  in  Table  1),  it  can  be  deter¬ 
mined  that  there  is  0.1  probability  of  project  cost  overrun  that  would 
exceed  24%  of  its  original  scheduled  cost  (see  Haimes  and  Chittister 
1993).  Thus,  the  conditional  expected  value  of  extreme  cost  overrun 
between  24%  and  30%  (or  extreme  cost  overrun  with  exceedance  prob¬ 
ability  that  is  below  0.1)  is  27%. 

COMPARATIVE  ANALYSIS  AMONG  CONTRACTORS 
An  important  activity  within  Phase  III  of  the  four-phase  acquisition 
process  is  understanding  the  reasons  and  the  genesis  for  the  variations 
of  the  estimates  among  the  contractors  and  explaining  these  differences 
on  the  basis  of  the  evidence  collected  through  the  Taxonomy,  the  inter¬ 
views,  and  other  ways. 

1.  The  contractor  knows  what  is  to  be  known  about  the  project. 

2.  The  contractor  does  not  know  what  is  to  be  known  about  the 
project. 
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Table  1. 

COMPARATIVE  TABULAR  CDF 


Fractile 

Project  Cost  Increase  (%) 

Customer 

Contractor  A 

Contractor  B 

0.00 

0 

0 

0 

0.25 

5 

10 

15 

0.50 

10 

15 

20 

0.75 

15 

20 

25 

1.00 

30 

50 

40 

3.  The  contractor  knows  and  is  aware  of  the  unknowns  and  uncer¬ 
tainties  about  the  project. 

4.  The  contractor  does  not  know  and  is  not  aware  of  the  uncertain¬ 
ties  surrounding  the  project. 

Of  course  this  knowledge  or  the  lack  thereof  is  not  absolute  and  the 
contractor’s  own  knowledge  may  be  at  various  levels  between  complete 
awareness  and  complete  ignorance.  This  discussion  assumes  that  no  gam¬ 
ing  is  taking  place.  Safeguards  must  be  developed,  however,  to  secure 
against  a  contractor  who  opts  to  game  the  system. 

With  these  and  other  comparisons  that  can  be  made  as  desired,  the 
customer’s  ability  to  assess  the  various  risks  and  thus  to  mitigate  and 
manage  them  is  greatly  enhanced.  For  example,  when  there  appears  to 
be  a  substantial  discrepancy  either  between  the  customer’s  and  one  of 
the  contractor’s  estimates  or  among  the  estimates  of  the  various  con¬ 
tractors,  the  customer  can  inquire  (if  legally  permissible)  about  evidence 
and  sources  of  these  variations;  otherwise,  the  customer  can  draw  con¬ 
clusions  on  the  contractor’s  estimation  capabilities  and  honesty.  The  use 
of  SEI’s  software  Risk  Taxonomy-Based  Questionnaire  [SEI  1993]  in 
conjunction  with  these  analyses  will  be  discussed  in  a  subsequent  sec¬ 
tion. 
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Table  2. 

SUMMARY  OF  RESULTS 


Customer 

Contractor  A 

Contractor  B 

Unconditional 
Expected  Value, 
«(•) 

11.25% 

17.50% 

20% 

Partitioning 

Point 

a  =  0.90 

a  =  0.90 

Corresponding 
Percent  of  Cost 
Increase 

x  =  24% 

x  =  38% 

x  =  34% 

Conditional 
Expected  Value, 
f4(.) 

27% 

44% 

37% 

The  methodology  advocated  in  this  paper  does  not  embrace  an 
adversarial  relationship  between  the  customer  and  the  prospective  con¬ 
tractors.  Project  cost  overrun  and  schedule  time  delay  are  not  assumed 
to  happen  necessarily  because  of  contractors’  conspiracy  or  malice. 
Rather,  the  premise  here  is  that  often  the  customer  and  the  contractors 
do  not  adhere  to  a  systemic  risk  assessment  approach  in  their  evaluation 
and  projection  of  software  nontechnical  risk,  and  the  unintended  result 
is  a  cost  overrun  or  delay  in  project  completion  schedule. 

The  use  of  the  Software  Risk  Taxonomy-Based  Questionnaire  devel¬ 
oped  by  the  Software  Engineering  Institute  constitutes  the  basis  for  this 
needed  systemic  risk  assessment  approach  (Carr  et  al.  1993).  The  SEI 
taxonomy  questionnaire  is  divided  into  three  major  parts:  Product  Engi¬ 
neering,  Development  Environment,  and  Program  Constraints. 

The  primary  focus  of  the  SEI’s  risk  identification  process  is  to  elicit 
known  and  unknown  risks  from  the  personnel  associated  with  the  project, 
e.g.,  administrative  and  technical  management,  development  engineers, 
proposal  team,  and  cost  estimators.  The  identification  process  consists 
of  a  taxonomy-based  questionnaire  and  a  method  for  conducting  inter¬ 
views  using  this  questionnaire.  This  enables  the  interviewers  to  probe 
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for  both  technical  and  nontechnical  risks  that  affect  the  project.  The 
information  that  is  gathered  from  the  interviews  can  be  grouped  and 
ordered  using  a  set  of  criteria  and  a  risk  ranking  and  filtering  method. 
The  strength  of  this  approach  is  that  the  process  is  repeatable  and  sys¬ 
tematic,  and  it  enables  the  analysis  and  comparison  of  data  from  mul¬ 
tiple  organizations.  The  analysis  and  comparison  of  risks  and  concerns 
coupled  with  the  extreme  events  information  will  provide  the  customer 
with  a  foundation  upon  which  to  make  more  informed  decisions  regard¬ 
ing  the  risks  in  the  cost  or  schedule  estimates. 

An  additional  benefit  of  the  analysis  is  that  customers  can  gain  valu¬ 
able  insight  into  the  contractors’  assumptions  and  the  depth  of  their 
understanding  regarding  the  requirements  and  technology  associated 
with  the  project. 

CONCLUSION 

Controlling  the  cost  and  time  schedule  of  major  projects  has  been  and 
continues  to  be  a  major  problem  facing  government  and  non-govern¬ 
ment  acquisition  managers.  Software  development  projects  are  no  ex¬ 
ception.  Because  of  the  close  influence  and  interaction  between  soft¬ 
ware  technical  and  nontechnical  risks  and  the  diverse  sources  and  causes 
that  constitute  the  driving  force  behind  these  risks,  the  acquisition 
manager’s  job  is  complicated.  One  of  the  major  premises  of  this  paper  is 
that  a  careful,  systemic,  and  analytically-based  process  for  contractor 
selection  is  imperative  for  the  prevention  of  major  risks  of  cost  overruns 
and  time  delays.  This  paper  proposes  such  an  acquisition  process.  The 
four-phase  process  can  be  best  viewed  as  a  framework  rather  than  as  a 
rigid  step-by-step  procedure.  The  obvious  limitation  in  the  scope  of  any 
single  paper  prevents  a  full  demonstration  of  each  of  the  four  phases  of 
the  proposed  acquisition  process.  The  three  sample  problems  should 
successfully  communicate  to  the  reader  the  mathematical  mechanics  as¬ 
sociated  with  Phase  I  and  the  construction  of  the  measure  of  risk  of 
extreme  events  in  Phase  II.  The  readers  who  are  more  familiar  with  the 
SEI  Taxonomy-Based  Questionnaire  will  be  able  to  relate  more  easily  to 
its  use  in  Phases  II  and  III.  Similar  statements  can  be  made  on  the 
familiarity  with  the  risk  ranking  and  filtering  method,  the  independent 
verification  and  validation  team,  and  with  other  methods  used  in  the 
proposed  acquisition  process.  As  a  framework,  the  discussion  in  this 
paper  must  be  construed  by  the  reader  as  the  beginning  of  a  dialogue 
toward  the  quantification  and  management  of  the  risks  of  cost  overruns 
and  time  delays  associated  with  software  development.  In  this  spirit,  we 
consider  this  paper  to  be  a  precursor  which  will  be  followed  by  others  in 
the  future.  The  expected  benefits  that  result  from  the  prevention  of 
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major  and  extreme  risks,  combined  with  the  low  expected  cost  of  early 
mitigation  strategies,  encourage  us  to  believe  that  this  area  is  worthy  of 
much  further  consideration. 
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